openarenafandomcom-20200216-history
Multithreaded map compiling
It is possible to take advantage of modern multi-core processors to compile maps much faster by using multithreading. Do you need this? Well, map compiling speed can really be improved a lot by it. If you are using Linux, map compiling uses a single thread by default, and you have to edit your "build/bsp" menu options in order to specify the number of threads to use. This can really speed up map compiling a lot! If you are using Windows, map compiling will use as many threads as your cores by default and usually you don't need to tweak it, unless you want to use fewer cores to have the computer more responsive for doing other tasks while compiling. If you wish to use multithread for .AAS (bot navigation) file creation, then you will have have to specify the number of threads manually, even on Windows. Overview Note: if you already know what how many cores your PC has got and you already decided how many threads you wish to use, you can skip this section. First of all, multithreading can only help you if your machine has got multiple "cores", at least "logical" ones. However, probably since around 2010, all PC processors have multiple "cores". If you have got more than one core, you may find useful to compile maps (q3map/q3map2) using more threads, in other words splitting the process into separate sections that can run at the same time on multi-core systems. This way, you should see higher CPU load in Task Manager/top while compiling, and you should be able to compile maps much faster than using a single thread, because each thread should work on a separate core (with a single thread, only one core would be used). So, choose the number of threads (1, 2, 3, 4, 5, 6...) depending from how many cores you have got and how many of them you want to use for this purpose: using all available cores should give you faster compiling. On the other hand, leaving at least one physical core "free" should allow you to do not experience slowdowns using the computer do to other things (e.g. playing a game or surfing the web) while you are compiling a map; it's your decision.'Another way to do not experience slowdowns while doing other things at the same time of compiling is to use tools as ''Task Manager or Process Explorer to mange process priority, by giving lower priority to the q3map/q3map2 process, or giving higher priority to the other process you are interested to. Howerver, this is not very comfortable, because you would have to manually repeat it every time q3map/q3map2 is invoked. You may wish to create various custom pre-sets using different numbers of threads, and choose each time which one to use depending from how much processing power you wish to have free for other applications. How to setup multithreaded compiling Q3MAP (map compiler that comes with the original Q3Radiant) and Q3MAP2 (map compiler that comes with GTK Radiant, with some extra features) are command line tools. The map editor launches q3map (or q3map2) with various command-line parameters, depending from the Bsp option you choose from the menu. Q3MAP and Q3MAP2 support the "-threads n" parameter to tell how many threads they have to use ("n" is the number of threads, e.g. 1, 2, etc.). * While running on Microsoft Windows OS, Q3MAP2 auto detects the number of total cores (physical + logical) your machine has, and automatically selects the number of threads to use accordingly. However you can use the "-threads" parameter to manually force the number of threads to use. * While running the compiler on GNU/Linux OS, instead, you have to manually specify the number of threads to use. * Also the old Q3MAP '''can automatically use a thread for each core it detects on Windows, if you don't specify the "-threads" parameter... but unfortunately some options in Q3Radiant "Bsp" menu do actually contain "-threads 1" by default, and so you will have to manually modify them if you want to use multiple threads. * BSPC tool (used to create .aas files) does not autodetect the number of cores, not even on Windows. You have to specify the number of threads if you wish to use multiple cores. See BSPC section for more infos. There is no advantage in trying to force more threads than the number of total cores you have. While using all your physical dual (or quad, or more) cores should speed up a lot, the effect of SMT (logical) cores may vary; anyway it seems SMT (e.g. HyperThreading) is quite good for map compiling, and using all of your cores should give you better results. If you wish, you can do tests compiling the same map specifying different number of threads, comparing the time needed.It may even be possible to completely disable HT technology from your system's BIOS/UEFI (this would make the system show half cores than before), and then do more tests. Other than the application in use, HT perfomances may also vary depending from the OS in use: if the Operating System is aware that some cores are physical and others are logical, it can better distribute the load; Microsoft reports that Windows is HT-Aware since Windows XP/Windows Server 2003, and we can guess later versions should be even more optimized. Some tests with the old q3map (Bsp_fullvis -light -extra from Q3Radiant, with tweaked threads number) on a Windows 7 machine with a quad-core 1st generation Intel Core i7 (8 total cores when HT is enabled; 4 total cores when HT is disabled), showed up the compiling of a test map in: * about 3 min 5 sec using 4 threads with HT disabled in BIOS * about 3 min 5 sec using 8 threads with HT disabled in BIOS * about 3 min 13 sec using 4 threads with HT enabled in BIOS * about 2 min 11 sec using 8 threads with HT enabled in BIOS. This seems to prove that compiling maps using as many threads as the total number of cores you see is good. For your information, the same compiling, using the option with the original "-threads 1" parameter, took 10 minutes 32 seconds. A note about the tests above: sometimes, q3map tended to give a crash error in the middle phase: in that case, we interrupted the compiling and re-launched it again, until the map compile completed without crashes. BSPC In a nutshell: just add -threads (change with the number of threads you wish) to your BSPC command line. Done. If you wish more detailed infos, continue reading this section. BSPC (the tool that creates "bot navigation" .AAS files) supports multiple threads, but it does not auto-recognize the numer of threads to use (not even on Windows), and will use one thread by default. You can however specify the number of threads to use by adding the -threads option, gaining compiling speed on multi-core machines. Just an example, if you lauch BSPC from a batch file that executes bspc -bsp2aas mymap.bsp, you can modify it to execute bspc -threads 2 -bsp2aas mymap.bspYou may place the ''-threads '' option at the end of the BSPC command line, if you prefer, it's the same. to run it with two threads instead of one. However, the time adavantage it gives is not as evident as with map compiling: test it and then decide... probably you can leave it without that option and still live happy (especially in case you notice enabling it causes BSPC to crash more often in your machine; however, switching to a different version of BSPC may fix the issue and allow you to use multithreading without crashing, hence allowing you faster .aas compiling.In a Windows 7 64 bit test, "BSPC version 2.0, Jun 21 2000 16:12:44 by Mr Elusive" -provided with Q3Radiant 202- did crash often with more than 1 thread, while "BSPC version 2.1h, May 6 2001 16:15:03 by Mr Elusive" -provided with GTKRadiant 1.5- and "BSPC version 2.1h, Oct 7 2012 15:47:47" -provided with GTKRadiant 1.6- did not.). You may also specify the -threads option to the batch file you use to optimize the .AAS file you previously created -the operation that makes the .aas file smaller- (example: bspc -threads 2 -aasopt mymap.bsp), but that operation is already so quick you don't need multithreading at all for it. By the way, it is possible (and probably advisable) to create aas file and optimize its size with a single command, using a command line like bspc -grapplereach -optimize -bsp2aas mymap.bsp Note: in this example, we also specified the "-grapplereach" option because it causes the "calulating reachability" phase to take some more time and you may take advantage of this to better compare BSPC performances when using different values for the "-threads" option. For your information, "-grapplereach" option is required for bots to use the grappling hook, if the server allows them to use it (bot_grapple 1). However in-game bot grapple support is incomplete and buggy since Q3A, and currently (OA 0.8.8) it has not been fixed yet. NetRadiant * 1) In NetRadiant 1.5 or NetRadiant Custom, go to "Build" -> "Customize..." menu. * 2) There, click on the entry of the list that you want to modify, to highlight it. * 3) Click on the line inside the "command line" section of the screen to highlight it, then click again to edit it. * 4) Add -threads N (where N is the number of threads you wish to use -usually the total number of logical "cores" of your machine, fewer for a more responsive machine for other tasks during compiling-) at the end of the line, then confirm with ENTER. Example: changing q3map2 -light -fast -super 2 -filter -bounce 8 "MapFile" into q3map2 -light -fast -super 2 -filter -bounce 8 "MapFile" -threads 8 or (nicer) q3map2 -light -fast -super 2 -filter -bounce 8 -threads 8 "MapFile" * 5) For the build options which include all passages (bsp, vis and light), add the option to all the three lines shown in the "command line" section. In other words, repeat step 4 as needed. * 6) Repeat steps 2-5 for all the build options you wish to set the number of threads to use (e.g. those you are used to use). * 7) Close the window with OK button. Done. You may instead wish to add new items in the "build" menu instead of modifying the existing ones, e.g. to have different options with different number of threads depending from how many cores you wish to leave free for other uses. In that case: * 1) Go to Build -> Customize... * 2) Click on the empty line at the bottom of the list to highlight it. Click again to edit it. * 3) Type the name of your new entry (you can copy-paste from an existing name and then change something), then press ENTER. * 4) Click on the first line (still emtpy) in the "command line" section to hightlight. Click again to edit. * 5) Write the command line. You can of course copy-paste from an existing one and then change something. Specify the desired number of threads. Confirm with ENTER. * 6) If you are making a build option which includes all passages: repeat steps 4 and 5 for the second and third line. Confirm with ENTER each time. * 7) Close the window with OK button. If you wish to delete an entry of the list, just highlight it and then press CANC. Or edit it, delete all characters to make it empty and then confirm with ENTER. Warning: there is no confirm request! GTKRadiant For GTK Radiant 1.5, see the section about NetRadiant, as it works the same. For GTK Radiant 1.6, proceed as follows instead. To edit an existing entry: * 1) Go to File -> Project Settings. * 2) Click on an entry of the "Menu commands" list, then click "Change" button. * 3) Modify the Command field to add in the "-threads N" option, for each stage. Replace N with the desired number of threads, of course. In this example, we modified a complete compile with the three passages (bsp, vis and light), so you can find three "-threads 4" highlighted in bold to show what we changed. ! "C:/GtkRadiant-1.6.6-20180422/x64/q3map2" -v # -game quake3 -fs_basepath "C:/Quake III Arena/" -meta '-threads 4' $ && ! "C:/GtkRadiant-1.6.6-20180422/x64/q3map2" # -game quake3 -fs_basepath "C:/Quake III Arena/" -vis -saveprt '-threads 4' $ && ! "C:/GtkRadiant-1.6.6-20180422/x64/q3map2" # -game quake3 -fs_basepath "C:/Quake III Arena/" -light -fast -patchshadows -samples 3 -bounce 8 -dirty -gamma 2 -compensate 4 '-threads 4' $ As you can guess, after "&&" a new command starts, so q3map2 program is invoked three times within a single menu entry. Being just an example, don't mind the fact that there GTKRadiant was configured for Quake 3 Arena instead of OpenArena. And of course, the paths will point to your actual GTK Radiant and game folders. * 4) Close with OK, OK. Done. If you prefer to create additional entries (like to have different pre-sets with different number of threads depending from your need to perform other tasks during compiling), you can use "Add..." button, then enter a "Menu text" and a "command" (which, as seen, can consist of three concatenated q3map2), then close with OK, OK. You can of course copy-paste the text from existing menu entries and then modify, instead of typing everything from scratch. Warning: when you create a new menu entry, or rename an existing one, remember that its "name" ("Menu text") has to begin with "bsp" (e.g. "bspMyTest", "bsp_MyTest", "bsp MyTest"...). If the name will not comply with this rule, you will lose your changes! It's possible to delete a menu entry using the apposite "Remove" button after selecting it. Tip: you can enlarge the "Edit Command" window by dragging its borders with the mouse. However you can also copy-paste its content into a plain text editor (like notepad), make your changes there and then copy-paste again into the window, if you prefer. Trivia: editing the bsp menu list through "project settings" window is how Q3Radiant should have worked, in case it didn't have a certain annoying bug with it. Q3Map2Build With Q3Map2Build (a small but useful tool that allows to compile a map -and optionally, also its .aas file- without launching a Radiant editor, with full control over compiling options), you can specify the number of threads to use by clicking on the "Options" buttons corresponding to each stage (BSP, VIS, LIGHT, BSPC). Such "Options" buttons are active only if you have chosen "Custom" for that stage. While in selected stage options window, just check "-threads" option and specify the desired number of threads in the apposite field, and click "Save". After you set options for all stages, build (compile) the map. Q3Radiant Note: please read all this section before trying to apply it: as later said, a bug in the editor may mess up your compiling options even if you didn't actually change them! In short, to avoid that bug, you will need to manually tweak a file called "quake.qe4" with a text editor (but it's worth the effort, as unfortunately some menu options did have "-threads 1" by default). Let's take the case we are using the old Q3Radiant editor, configured for the original Quake III Arena game, on Microsoft Windows''Hint:'' it looks like that Q3Rradiant (or Q3MAP, which is invoked by Q3Radiant when compiling) is more stable if run as administrator: if you encounter crashes while running it without administrative privledges, try launching Q3Radiant as administrator.. In the "Bsp" menu, you can find the bsp_FullVis (light -extra) option. This is probably the one that takes more time to compile, used also for the "final" map, and unfortunately this menu option specifies "-threads 1" parameter,While without it, you may achieve noticeably lower compiling time, if you have multiple processors, multiple cores or HyperThreading. so it is the one we will tweak. You may notice great time benefits after the tweak. Go to "File" -> "Project settings" menu. In the "Menu commands" section, select bsp_FullVis (light -extra) line and click "Change...". In the window that opens, select all the text in the "Command" field, and copy it to your clipboard (CTRL+C or right button, copy), then click on "Cancel", to be sure we will not damage the original option. Now click on "Add...", and paste from your clipborard into the "Command" field (CTRL+V or right button, paste). Modify from -threads 1 to -threads 2 (or other number, e.g. 3, 4, 5, 6...) in the two places where the "-threads" option appears, and optionally add a third one in the first part of the command if you like (as you can see, this menu entry invokes q3map program three times -a new command starts after "&&"-, and only two of them have the dreaded "-threads 1" option). Enter bsp_FullVis (light -extra) Multithread (or whatever you prefer, but anyway the name has to begin with bsp, and it should not contain the " character) in the "Menu text" field, then click OK.Alternatively, when you click on "Modify..." after selecting the original menu option, edit the "threads" parameters there and MODIFY the "menu text", then click OK: this will cause the program to automatically create a new menu option, without the need of copy-pasting (we suggest to do not apply any change directly to the original menu options, but to create new options, because you may risk to ruin them, also due to a Q3Radiant bug that may "cut" the command line!). You should have started from something similar to this (of course your Q3 installation path may vary): ! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -vis -threads 1 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -light -extra -threads 1 $ And you should have got something like this (in case you want 2 threads): ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -threads 2 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -vis -threads 2 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -light -extra -threads 2 $ Still in the "Project Settings" windows, select the newly created option and click "Change...". If everything is still as you set a minute ago, you are ready and you can go compiling your map... but if the line in the "Command filed" appears incomplete (where many options misteriously disappeared), e.g. like this: ! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map you enconountered a bug of the editor (this is another reason we suggest to never modify the original menu options) and thus you have to manually modify the file that contains the menu to fix the menu option (compiling would not run correctly without the missing part of the command line). In this case, close Q3Radiant. Then go to your Quake III Arena installation folder (if you have multiple installations of the game, the one you use with Q3Radiant), then enter the "BASEQ3" folder, and the "scripts" folder in it. There, use a plain text editor (such as notepad) to edit the "quake.qe4" file. Locate the line that contains your newly created menu option (in our example, bsp_FullVis (light -extra) Multithread). As you can guess, the format of a line is simply "menu text" "command". Correct the command line part, copy-pasting from the original menu option and changing the value in the two places of the line where the ''-threads'' option is. Save and close the file, and start Q3Radiant again. At this point, if you select the newly created option from the Bsp menu, you should notice your map compile in less time. Of course, you may have directly edited the quake.qe4 file and added a line (somewhere between the { and } lines) for the new menu option, if you prefer. Also in this case, remember that the name of the new menu option has to begin with bsp and should not contain the " character (that is instead used to separe the menu option name from the command line). Some hints: * Also considering the editor bug, you may want to make a backup copy of that quake.qe4 file before starting to make any changes to it through Q3Radiant or through text editor. * In our example, we specified the number of threads we want. Anyway you can set q3map to automatically use the same number of threads as the total number of CPU cores Windows finds, by removing the "-threads n" option from the desired line in quake.qe4 file, in the two places of the line where it appears. * You may also wish to edit quake.qe4 file with a text editor to quickly update all the paths in case you changed your Q3A install location. But note that the "Prefab path" string in "Q3Radiant Preferences" window is stored in some other file. Notes See also * Mapping * Mapping resources & tutorials * Speed up visibility calculations Category:Development Category:Mapping